Bloom Filters, Adaptivity, and the Dictionary Problem

The Bloom filter—or, more generally, an approximate membership query data structure (AMQ)— maintains a compact, probabilistic representation of a set S of keys from a universeU . An AMQ supports lookup, and possibly insert and delete operations. If x ∈ S, then lookup(x) returns “present.” If x 6∈ S, then, lookup(x) may return “present” with probability at most ε, where ε is a tunable false-positive probability, and such an x is called a false positive of the AMQ. Otherwise lookup(x) returns “absent.” AMQs have become widely used to accelerate dictionaries that are stored remotely (e.g., on disk or across a network). By using an AMQ, the dictionary needs to access the remote representation of S only when the AMQ indicates that the queried item might be present in S. Thus, the primary goal of an AMQ is to minimize its false-positive rate, so that the number of unnecessary accesses to the remote representation of S can be minimized. However, the false-positive guarantees for AMQs are rather weak. The false-positive probability of ε holds only for distinct or randomly chosen queries, but does not hold for arbitrary sequences of queries. For example, an adversary that chooses its queries based on the outcomes of previous queries can easily create a sequence of queries consisting almost entirely of false positives. Even simply repeating a randomly chosen query has an ε chance of producing a sequence entirely of false positives. In this paper, we give adaptive AMQs that do have strong false-positive guarantees. In particular, for any fixed ε, our AMQs guarantee a false-positive rate of ε for every query and for every sequence of previously made queries. Furthermore, our adaptive AMQ is optimal in terms of space (up to lower order terms) and complexity (all operations are constant time). This research was supported in part by NSF grants CCF 1114809, CCF 1217708, CCF 1218188, CCF 1314633, CCF 1637458, IIS 1247726, IIS 1251137, CNS 1408695, CNS 1408782, CCF 1439084, ccf-bsf 1716252, CCF 1617618, IIS 1541613, and CAREER Award CCF 1553385, as well as NIH grant 1U01CA198952-01, by the European Research Council under the European Union’s 7th Framework Programme (FP7/2007-2013) / ERC grant agreement no. 614331, and by Sandia National Laboratories, EMC, Inc, and NetAPP, Inc. ∗Stony Brook University, Stony Brook, NY 11794-4400, USA. Email: {bender,shiksingh}@cs.stonybrook.edu. †Rutgers University, Piscataway NJ 08855, USA. Email: [email protected]. ‡Queens College, CUNY, New York, USA. Email: [email protected]. §VMware Research, Creekside F, 3425 Hillview Ave, Palo Alto, CA 94304. Email: [email protected] ¶IT University of Copenhagen, Copenhagen, Denmark. Email: [email protected].